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SOURCES OF INFORMATION FOR RESPONSES TO MULTIPLE-CHOICE 
ITEMS ON READING COMPREHENSION TESTS AND THE EFFECTS OF 
CONTENT WORDS AND FUNCTION WDRDS IN SUPPLYING INFORMATION FOR 
THE CORRECT COMPLETION OF MULTIPLE-CHOICE ITEMS WERE STUDIED 
IN AN EXPERIMENT AT CAMPBELL COLLEGE. PARAGRAPHS WITH 
ACCOMPANYING MULTIPLE-CHOICE ITEMS SAMPLED FROM READING TESTS 
LISTED IN BUROS SIXTH YEARBOOK WERE SELECTED AS THE TEST 
PASSAGES. SUBJECTS WERE 147 SOPHOMORES ENROLLED IN 
INTRODUCTORY PSYCHOLOGY COURSES AND WERE RANDOMLY ASSIGNED TO 
SEVEN TREATMENTS INVOLVING THE BLACKING OUT OF (1) ALL OF THE 
READING PASSAGE, (2) ALL NOUNS, VERBS, AND ADJECTIVES, (3) 

ALL FUNCTION WORDS, (4) NONE OF THE READING PARAGRAPH, (5) 10 
PERCENT OF THE NOUNS, VERBS, AND ADJECTIVES, (6) 50 PERCENT 
OF THE NOUNS, VERBS, AND ADJECTIVES, AND (7) 90 PERCENT OF 
THE NOUNS, VERBS, AND ADJECTIVES. A TREND ANALYSIS OF THE 
EFFECT OF BLACKING-OUT INCREASING PERCENTAGES OF NOUNS, 

VERBS, AND ADJECTT'VES AND COMPARISONS BETWEEN TREATMENTS 1 
AND 3 AND BETWEEN TREATMENTS 3 AND 4 WERE MADE. IN THE 
DISCUSSION OF RESULTS, IT WAS CONCLUDED THAT READING 
COMPREHENSION TESTS WERE HIGHLY DEPENDENT ON EXAMINEE 
CHARACTERISTICS WHICH OFTEN HAD LITTLE TO DO WITH THE READING 
TASK BEING EXAMINED. APPARENT REASONS FOR THIS ARE PROPOSED 
AND ANALYZED. TABLES AND REFERENCES ARE INCLUDED. THIS SPEECH 
APPEARS IN THE 1967 PROCEEDINGS OF THE AMERICAN PSYCHOLOGICAL 
ASSOCIATION, DIVISION 15. <LS) 
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Sources of information for responses to reading test items 

Wendell W. Weaver 
The University of Georgia 
and A. C. Bickley 
C ampbell College 

The study dicotomizes the source of information in responding to 
multiple-choice items on reading comprehension tests into "learnings 
brought to the test situation" by the examinee and "information derived 
from the reading display itself." The effects of content words and function 
words in supplying information for the correct completion of multiple- 
choice items is also examined. 

By using a modification of a black-out technique it was found that 
some reading test items can be answered as well without reading a sup- 
posedly relevant paragraph as when the paragraph is read. Apparent rea- 
sons for this effect are proposed and analyzed. 
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Sources of Information for Responses to Reading Test Items 

Wendell IV. Weaver 



The University of Georgia 
and A. C. Bickley 
Campbell College 

It is not uncommon to find in the reading literature the question , 
"What do reading tests really measure?" (Kingston, 1965). A cursory look 
at the task imposed by a reading comprehension test indicates that the 
person taking the test is to read the paragraph and then answer the mul- 
tiple-choice questions which follow. The assumption is that the antece- 
dent behavior of reading the paragraph is necessary (even if not suffi- 
cent) for answering the test items correctly. That a prior perusal of a 
graphic display is a requirement for reading to have occurred is indicated 
by the definition of reading in English and English (1958, p.442). 

Reading is defined as, "the perception of written, printed, or engraved 

symbols constituting a communication " One of the purposes of this 

study is to attempt to clarify empirically the source of semantic infor- 
mation as measured by reading comprehension tests. 

Language has been divided into two categories by numerous linguists. 
Hockett (1963) has pointed out that a characteristic of language which 
has been called a universal of language is the dicotomy between grammatical 
and non- grammatical meaning. Non-grammatical meaning is synonymous with 
the term "conceptual information." Rankin (1957) has shown that conceptual 
information is dependent upon the nouns and verbs in the reading passage 
more than is grammatical meaning and that grammatical meaning is more 
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closely connected with function words than it is with nouns and verbs 
(often called ''lexical'’ words). Another purpose of this study is to 
examine the relationship between grammatical and non -grammatical features 
of language with regard to their effects upon conceptual information as 
measured by the multiple-choice items on reading comprehension tests. 

Method 

The black-out technique was developed by Holland and Kemp (1965) 
for the study of programmed instruction. They were attempting to deter- 
mine which frames were, and which frames were not, giving information 
which allowed the response frame to be completed correctly. It was 
found that many frames had cues to the correct answer that were unrelated 
to the topic which the programmers intended to teach. These experimenters 
chose words to blackout that they judged to contain crucial information 
for answering the questions asked about the material. 

It was hoped that this technique could be used to analyze reading 
comprehension tests. However, it soon became apparent that many reading 
test items were not dependent on specific locations or specific words in 
the reading display. Therefore, in this study percentages of content 
words were blacked out. 

The materials were paragraphs sampled from reading tests listed in 
Buros ' (1965) sixth yearbook. The paragraphs taken from the different 
tests were selected arbitrarily with length and number of questions per- 
taining to the paragraphs the only constraining criteria. 

The Ss were 147 sophomores, randomly selected from 240 sophomores 
enrolled in courses in introductory psychology at Campbell College and 
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randomly assigned to seven treatments, as follows: 

1. All of the reading passage blacked out 

2. All nouns, verbs, and adjectives blacked out 

3. All function words blacked out 

4. None of the reading paragraph blacked out 

5. 10% of the nouns, verbs, and adjectives blacked out 

6. 50% of the nouns, verbs, and adjectives blacked out 

7. 90% of the nouns, verbs, and adjectives blacked out 

The questions pertaining to the reading passages were not altered 
in any way and the subjects were instructed to answer all the questions 
from whatever information was available to them in their particular 
reading passages. 

Results 

Two types of orthogonal comparisons were planned (See Table 1) . 

One set of comparisons was whether a significant difference was present 
between the means of treatments one and three and between treatments three 
and four. For the former comparison, a non-significant difference would 
indicate that function words add nothing to conceptual meaning; for the 
latter, a difference would indicate the function words, on their own, 
add conceptual meaning. 

The other set of comparisons was a trend analysis of the effect of 
blacking-out increasing percentages of nouns, verbs, and adjectives. It 
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was assumed that a deletion of 10%, 50% and 90% of lexical items, while 
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leaving the function words intact, would be related linearly to the number 
of multiple-choice items answered correctly. 

Orthogonal planned comparisons of means and a trend analysis were 
computed. There was a significant difference in the number of correct 



Insert Table 2 about here 



answers to multiple-choice items on a passage when the paragraph on 
which the items (supposedly) depended was intact, as opposed to correct 
answers to the same items when function words were blacked out and only 
lexical words remained. There was no significant difference in multiple- 
choice item responses when function words alone were compared to a com- 
plete deletion of the paragraph from which the items were drawn (See 
Table 2). 

There is a significant linear trend with a nonsignificant quadratic 
component. While not statistically appropriate in this analysis, adding 
the 100% deletion treatment and the 0% treatment to the series increases 
the F-ratio. 

Discussion 

The relationship of function words and lexical words to the com- 
pletion of multiple-choice items on reading tests is a complex one. For 
this population, function words alone, supplied no conceptual information. 
Lexical words alone, supplied twice as much information as no paragraph 
at all (when the guessing score was subtracted) . Function words in com- 
bination with lexical words (the normal language situation) have a large 
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effect on the completion of the multiple-choice items, but by no means 
the unique effect ordinarily assumed by users of reading test. 

The effect of removing increasing proportions of lexical words 
with the rest of the context intact is linear (See Table 2). This implies 
that the conceptual information needed to reduce uncertainty about reading 
test items is scattered widely throughout a passage. A reduction of nouns, 
verbs, and adjectives reduces available conceptual information proportionali- 

iy. 

The general impression of the results is that there are at least 
two relatively distinct sources of information for the completion of 
multiple-choice items in reading comprehension tests. One source is the 
conceptual information the subject brings to the testing situation about 
the topics which the test samples. The previously learned information 
apparently allows the subjects to correctly respond to test items on the 
basis of greatly reduced cues in the reading passages, or in the absence 
of a reading passage, on the basis of cues in the test items alone. Ss 
who had no reading passage to aid in answering the item, nevertheless, 
correctly completed 67% as many items as Ss with all the reading passage. 
Perhaps this was accomplished in one or more of the following ways: 

1. £s knew the correct answer from past learning 

2. £s eliminated distractors which were not likely in any reading 
situation 

3. Ss used information from preceding items to reduce uncertainty 
about succeeding items. 
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It was possible, using the data gathered in this study, to make an 
informed guess as to whether or not the Ss used information from preceding 
items to reduce uncertainty about items which followed. There were five 
sets of three sequential items each, e.g. 21, 22, 23, for which there 
were high proportions of correct responses under the complete blackout 
treatment. Other than these five sets, only two single items had over 
50% correct responses under the complete blackout treatment. 

A modification of Guttman's scaling technique was used to analyze 
the responses. In all but one of the five sets proportionality was greater 
than .90--in that case it was .82. While nothing more than a highly sug- 
gestive relationship should be inferred from these "rough-and-ready" 
operations, indications are that a potential major source of cues to con- 
ceptual information on a paragraph reading test is the information derived 
from preceding items concerning a particular topic. 

The results of this study indicate that reading comprehension tests 
are highly dependent on examinee characterictics which often have little 
to do with the reading task the examiner assumes he is presenting. 

Reading tests are measuring past learning, word association, irrelevance 
of distractors, and "item conceptual -information constraints," as well as 
the person's ability to answer multiple-choice items directly from cues 
in the reading display. The sources of variation are so confounded that 
two, or more, factors could be hidden here, and one would never know. 

Much of the confounding could be reduced by changes in methods of se- 
lecting items. 
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Table 1 

Weights of Planned Comparisons 
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Table 2 

Significance Levels of Orthogonal Comparisons 



Variation 

source 


Mean 

difference 


Degrees 

freedom 


Mean 

t square 


F 


Comparison 1 


3.95 b 


126 


4 . 76 a 




Comparison 2 


1.24 b 


126 


1.49 




Comparison 3 




1 


247. 


15. a 


Deviations 




1 


7. 




Within Grps 




60 


15. 





Significant at .05 level. 
Standard error .83. 



